Vehicle display system with user input display

ABSTRACT

A display system for a vehicle includes an interior camera disposed in an interior cabin of a vehicle and having a field of view interior of the vehicle that encompasses an area at or near a user input for an accessory, with the user input being accessible by a driver of the vehicle when the driver is normally operating the vehicle. A display device is viewable by the driver when the driver is normally operating the vehicle. The display device is operable to display images derived from image data captured by said interior camera. The display device displays images depicting the driver&#39;s hand as the driver&#39;s hand approaches the user input for the accessory for actuating the user input to adjust or control the accessory.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application claims the filing benefits of U.S. provisionalapplications, Ser. No. 62/355,459, filed Jun. 28, 2016, and Ser. No.62/344,528, filed Jun. 2, 2016, which are hereby incorporated herein byreference in their entireties.

FIELD OF THE INVENTION

The present invention relates generally to a vehicle vision system for avehicle and, more particularly, to a vehicle vision system that utilizesone or more cameras at a vehicle.

BACKGROUND OF THE INVENTION

Use of imaging sensors in vehicle imaging systems is common and known.Examples of such known systems are described in U.S. Pat. Nos.5,949,331; 5,670,935 and/or 5,550,677, which are hereby incorporatedherein by reference in their entireties. Head up displays (HUD) are alsoknown and display images at a windshield area of the vehicle. Examplesof known HUD systems are described in U.S. Pat. Nos. 8,953,247 and/or8,427,751, which are hereby incorporated herein by reference in theirentireties.

SUMMARY OF THE INVENTION

The present invention provides a vision system or imaging system ordisplay system for a vehicle that utilizes one or more cameras(preferably one or more CMOS cameras) to capture image datarepresentative of images interior the vehicle, and provides display (forviewing by the driver) of the driver's hand (or portions thereof oriconistic representations thereof) as the driver's hand approaches andcontacts a user input to control or adjust an accessory of the vehicle.The display may comprise a head up display or HUD (that may projectdisplayed images at the windshield area forward of the driver so thedriver can readily view the displayed images without taking his or hereyes off the road). The system of the present invention thus providesdisplay of the driver's hand to assist the driver in controlling oradjusting a user input of an accessory so that the driver can achievethe desired control or adjustment without having to look downward towardthe user input at the instrument panel or dashboard or console of thevehicle.

These and other objects, advantages, purposes and features of thepresent invention will become apparent upon review of the followingspecification in conjunction with the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a plan view of a vehicle with a vision system thatincorporates cameras in accordance with the present invention;

FIG. 2 shows a vehicle interior cabin with touch screen, knobs at thecentral cluster, a camera attached at the inner center rearview mirror,a camera integrated into the instrument cluster, and a HUD vision image25 projected onto the windshield;

FIG. 3 shows a vehicle interior cabin similar to FIG. 2, with a driverat the driver seat using the touch screen at the central cluster andhaving his hand visible at the HUD image in an augmented way;

FIG. 4 shows the HUD image of FIG. 3, showing a hand stylized over astylized menu in accordance with the present invention;

FIG. 5 shows a HUD vision image in accordance with the presentinvention, which may be generated as a visible image to the driver asshown in FIGS. 2, 3 and 7, showing the driver's pointing finger as anarrow hovering over the letters he or she is selecting;

FIG. 6 shows a HUD vision similar to FIG. 5, where the driver's fingertips that she or he is putting on a touch screen are shown augmented ascircles in the HUD view; and

FIG. 7 shows a vehicle interior cabin similar to FIGS. 2 and 3, with adriver at the driver seat controlling control knobs at the centralcluster and having his hand visible naturally at the HUD image butsemi-transparent so that also the knob underneath the hand is visible.

LEGEND

-   -   10 vehicle    -   12 vision system    -   14 a, 14 b, 14 c, 14 d cameras    -   18 processor    -   16 display    -   20 rearview mirror assembly    -   22 camera in mirror or elsewhere    -   23 camera disposed near the central instrument cluster    -   24 head up display device    -   25 head up display image on windshield    -   26 instrument cluster    -   27 central cluster    -   28 knobs    -   29 touch screen    -   30 driver compartment    -   31 left side window    -   32 windshield    -   33 touch screen display content    -   34 driver

DESCRIPTION OF THE PREFERRED EMBODIMENTS

A vehicle vision system and/or driver assist system and/or objectdetection system operates to capture images interior and/or exterior ofthe vehicle and may process the captured image data to display imagesand/or to detect objects at or near the vehicle and in the predictedpath of the vehicle, such as to assist a driver of the vehicle inmaneuvering the vehicle in a rearward direction. The vision systemincludes an image processor or image processing system that is operableto receive image data from one or more cameras and provide an output toa display device for displaying images representative of the capturedimage data. Optionally, the vision system may provide a top down orbird's eye or surround view display and may provide a displayed imagethat is representative of the subject vehicle, and optionally with thedisplayed image being customized to at least partially correspond to theactual subject vehicle.

Referring now to the drawings and the illustrative embodiments depictedtherein, a vehicle 10 includes an imaging system or vision system 12that includes at least one exterior facing imaging sensor or camera,such as a rearward facing imaging sensor or camera 14 a (and the systemmay optionally include multiple exterior facing imaging sensors orcameras, such as a forward facing camera 14 b at the front (or at thewindshield) of the vehicle, and a sideward/rearward facing camera 14 c,14 d at respective sides of the vehicle), which captures images exteriorof the vehicle, with the camera having a lens for focusing images at oronto an imaging array or imaging plane or imager of the camera (FIG. 1).The cameras may be arranged so as to point substantially or generallyhorizontally away from the vehicle. The lens system's vertically openingangles α may be, for example, around 180 degrees, and the horizontallyopening angles β may be, for example, around 180 degrees. The visionsystem 12 includes a control or electronic control unit (ECU) orprocessor 18 that is operable to process image data captured by thecameras and may provide displayed images at a display device 16 forviewing by the driver 33 of the vehicle (although shown in FIG. 1 asbeing part of or incorporated in or at an interior rearview mirrorassembly 20 of the vehicle, the control and/or the display device may bedisposed elsewhere at or in the vehicle).

In the illustrated embodiment of FIGS. 1-3 and 7, the vision systemincludes an interior monitoring camera 22 having an interior field ofview to capture images or image data of and track or monitor theposition of the driver's hands and/or head and/or the direction that thedriver's eyes are looking. The camera 22 may be disposed at or in themirror head of the mirror assembly (and may be adjusted with adjustmentof the mirror head) or may be disposed elsewhere within the interiorcabin of the vehicle, with its field of view encompassing an area thatis typically occupied by a driver of the vehicle when the driver of thevehicle is occupying the driver's seat of the vehicle and normallyoperating the vehicle. The camera 22 or optionally another camera 23 mayalso be disposed near the instrument cluster 26, or the central cluster27, having the input interfaces, such as the buttons, knobs 28 or touchscreen 29 or touch screens, in the field of view for capturing theoccupants, especially the driver's hand. In the illustrated embodiment,camera 23 is integrated into the instrument cluster and camera 22 isattached at the interior rearview mirror. Responsive to image processingof image data captured by the interior monitoring camera, the system maydetermine the user's hand or finger's pose and/or the finger's positionand optionally, but not necessarily, also the gestures (by that asequence or hand or finger poses) made by the driver (or otheroccupant), and may control one or more of the cameras according to thedetermined position or gesture or movement.

In gesture recognition systems, learning and calibration modes for userspecific gesture profile generation/improvement are specified. The userspecific gesture profile generation/improvement requires active effortsof driver input, either by passing a calibration procedure or by givingredundant input by voice to confirm or disconfirm user gesture inputsrecognitions or to strengthen the its confidence level.

Typically, such gesture recognition or driver monitoring systems employone or two cameras, such as one or two infrared (IR) or near IR camerasand an IR or near IR illumination source. Some of such systems use timeof flight (TOF) cameras for capturing 3D images or sequences. Anassociated control and image processor or image processing system mayprocess captured image data and, responsive to such image processing,discriminate a driver's (or user's) hand (and eventually arm andfingers) position.

The present invention utilizes an interior monitoring camera, or nearinfrared camera or similar detecting device, in the area of the centralcluster (optionally at a roof area and looking downward toward theinstrument panel and user inputs located thereat) for capturing thedriver's hand and input devices such as the radio volume knob. Thecamera's live images may be presented as part of the (real time) imagecontent 25 of a head up display device 24 (HUD), or of a display screendisposed at an area that is readily viewable by the driver when drivingthe vehicle so that the driver does not have to take his or her eyes offthe road while driving and viewing the display screen. By that, thedriver will be able to direct his or her hand toward the inputinterfaces properly without looking at the input device (typicallylocated toward the center of the vehicle and not in the driver's directfield of view when looking forward through the windshield) but insteadbased on viewing the augmentation in front of him or her. Optionally,another part of the HUD image content may be conventional content (suchas showing the current speed limit as a sign symbol such as shown inFIGS. 3, 4 and 7).

The driver's hand 35 may be shown as a real image directly captured bythe camera, whereby the driver can readily view the displayed realimages as the driver's hand approaches the accessory or controls,especially in case where the control button, knob or lever is a physicalone, not a soft-button or touch sensor. In this case, the realaccessories or controls may be shown as real images as well. Optionally,the accessories or controls may be shown in a stylized way, and thebuttons may be augmented or stylized as being under the approaching handor finger, preferably in the spatial orientation that the real hand orfingers are against the real accessories, controls or soft buttonposition. For example, the HUD may display the driver's pointing fingertogether with the copy or avatar of a touch screen showing alphanumericsoft buttons of a navigation system input interface for entering cityand street names intuitively, and by that stress free, since the fingermovements will be directly reflected within the HUD image in front ofthe driver.

Optionally, the driver's hand may be shown in a stylized way, such as anavatar with the outlines of the hand (such as shown in FIGS. 2 and 3) orthe finger tips may be shown as circles as shown in FIG. 6 (or othericonistic representations of portions of the hand or finger tips),optionally in a contrasting color and/or in adapted brightness towardsthe background so that the driver can readily view and conceive thedisplayed images or outlines in all outside lighting conditions. In theexample of FIG. 6, the driver is rotating a navigation system's map, sothat the northeast direction is currently at the top of the displayedimage. Optionally, a pointing finger may be augmentedly displayed in theHUD view 25 as an arrow (or other iconistic representations) when beingat an display area at which different choices or selections areprovided, such as for selecting (soft-) roll boxes, grabbing a (soft-)handle, or such as shown in FIG. 5 when clicking (alphanumeric or other)letters, types or characters.

Optionally, the system may show touch screen control functions forcontrolling a virtual camera view of a surround vision system with3D-turning, scroll and zooming functions, soft button highlightingfunctions based on finger approximation, remote parking space [view]selection by touch screen, gesture control for easy menu navigation anda gesture input locking function for preventing unwilling gesture input,such as by utilizing aspects of the systems described in U.S.Publication Nos. US-2015-0022664, which is hereby incorporated herein byreference in its entirety. Optionally, the system may show surround viewvision system views at the HUD vision image 25.

Optionally, the driver's hand may be displayed or augmented partially orhalf transparent, especially when covering essential display contentunderneath such as the label of a button. This may also be done whendisplaying real knobs or buttons covered by an image (or avatar) of thedriver's hand such as shown in FIG. 7 or an augmented image of an arrowor hand such as shown in FIG. 4. The image information of the buttons orknobs hidden by the hand may come from a data storage, which may be partof the image processing system according the invention, as non-real time(and non-true) images for being blended or merged with the real timeimages of the hand, such as captured by camera 22 or 23 (or other camerain the vehicle).

In applications where the input device is a touch pad or touch screen,the touch pad or screen is able to sense the finger tips approaching andtouching very precisely (by its sensor nature). The touch screen ortouch pad sensing input may optionally be fused with the data of thecamera or cameras or taken alone as a control output for the hand orfinger augmentation. In case the input device is a touch screen, thescreen content or parts of it or a similar content may by shown as well,together with the hand, the outline of the hand or the finger tips. Atouch command may be stylized different than the fingers sliding overthe screen surface, a finger just resting on the surface or beingelevated from the surface. For generating stylized hand or finger tips,there may be a suitable image processing algorithm utilized similar tothose known from hand gesture capturing.

The camera and the display device or HUD may be activated to captureimage data and to display images responsive to the driver's handapproaching or touching an accessory control, whereby, upon the driver'sfirst touch at the user input or accessory control, the system isactivated to display images to assist the driver in further controllingor adjusting the accessory. Optionally, the camera may capture imagedata and an image processor may process the captured image to determinewhen the driver moves his or her hand towards an accessory control,whereby the system may automatically operate to display the images atthe display device for viewing by the driver while the driver isnormally operating the vehicle. The system may activate the displaydevice responsive to the driver's hand approaching selected accessories,such as HVAC controls, radio controls, navigation system controls, atouch screen, and/or any other accessories (or controls for accessories)typically disposed at or near the instrument panel or console or mirrorassembly of the equipped vehicle.

Although the system according the invention includes aspects of gesturecontrol, the present system has the advantage that no specific gestureset has to be trained to be interpreted as a command that the actualdriver has to know to operate the controls. No mix in gesture controlwith physical (direct) controls (such as turning a physically presentcontrol knobs) takes place, and by that an unfamiliar user is able tocontrol all desired functions as well as familiar functions, whereby theuser can then adjust controls faster and is less distracted due tohaving the visual aid.

The HUD vision and system of the invention may have a haptic feedbackwithin its input devices, accessories or controls, such as by utilizingaspects of the systems described in U.S. Publication Nos.US-2015-0022664 and US-2015-0277735, which are hereby incorporatedherein by reference in their entireties. The haptic feedback mayfeedback each user entry. When hovering with an arrow or finger pointerover view items there may be a haptic feedback, possibly according thevision's relief. The feedback may simulate a grid snapping when turning,scrolling or scaling a view. The system may feedback each scrollingstep, when scrolling through a menu or tool box. Optionally, the HUDimage displayed may change color or otherwise highlight a touched oractuated input to visually indicate that the user has successfullyactuated a selected input. The vision may utilize the letterrearrangement responsive to the user's finger widths, such as also shownin U.S. Publication No. US-2015-0277735.

Head and face direction and position tracking systems and/or eyetracking systems and/or gesture recognition systems may find use (oradditional use) in supporting or adding to other systems. Such head andface direction and/or position tracking systems and/or eye trackingsystems and/or gesture recognition systems may utilize aspects of thesystems described in U.S. Pat. No. 7,914,187 and/or U.S. PublicationNos. US-2016-0137126; US-2015-0352953; US-2015-0296135; US-2015-0294169;US-2015-0232030; US-2015-0022664; US-2015-0015710; US-2015-0009010and/or US-2014-0336878, which are hereby incorporated herein byreference in their entireties.

The head up display device or system may utilize aspects of the displaysystems described in U.S. Pat. Nos. 9,620,042; 9,405,120; 8,953,247;8,427,751 and/or 7,253,723, and/or U.S. patent application Ser. No.15/484,293, filed Apr. 11, 2017 (Attorney Docket MAG04 P-3003), Ser. No.15/432,950, filed Feb. 15, 2017 (Attorney Docket MAG04 P-2946), and/orSer. No. 15/364,354, filed Nov. 30, 2016 (Attorney Docket MAG04 P-2894),and/or U.S. provisional application Ser. No. 62/348,459, filed Jun. 10,2016, which are hereby incorporated herein by reference in theirentireties.

Optionally, the display of the vision system may utilize aspects of thevideo display systems described in U.S. Pat. Nos. 5,530,240; 6,329,925;7,855,755; 7,626,749; 7,581,859; 7,446,650; 7,338,177; 7,274,501;7,255,451; 7,195,381; 7,184,190; 5,668,663; 5,724,187; 6,690,268;7,370,983; 7,329,013; 7,308,341; 7,289,037; 7,249,860; 7,004,593;4,546,551; 5,699,044; 4,953,305; 5,576,687; 5,632,092; 5,677,851;5,708,410; 5,737,226; 5,802,727; 5,878,370; 6,087,953; 6,173,508;6,222,460; 6,513,252 and/or 6,642,851, and/or U.S. Publication Nos.US-2012-0162427; US-2006-0050018 and/or US-2006-0061008, and/orInternational Publication Nos. WO 2010/099416; WO 2011/028686; WO2012/075250; WO 2013/019795; WO 2012/075250; WO 2012/145822; WO2013/081985; WO 2013/086249 and/or WO 2013/109869, and/or U.S.Publication No. US-2012-0162427, which are hereby incorporated herein byreference in their entireties.

The system includes an image processor operable to process image datacaptured by the camera or cameras, such as for detecting objects or theuser's hand or the like in the field of view of one or more of thecameras. For example, the image processor may comprise an imageprocessing chip selected from the EyeQ family of image processing chipsavailable from Mobileye Vision Technologies Ltd. of Jerusalem, Israel,and may include object detection software (such as the types describedin U.S. Pat. Nos. 7,855,755; 7,720,580 and/or 7,038,577, which arehereby incorporated herein by reference in their entireties), and mayanalyze image data to detect vehicles and/or other objects.

For example, the vision system and/or processing and/or camera and/orcircuitry may utilize aspects described in U.S. Pat. Nos. 9,233,641;9,146,898; 9,174,574; 9,090,234; 9,077,098; 8,818,042; 8,886,401;9,077,962; 9,068,390; 9,140,789; 9,092,986; 9,205,776; 8,917,169;8,694,224; 7,005,974; 5,760,962; 5,877,897; 5,796,094; 5,949,331;6,222,447; 6,302,545; 6,396,397; 6,498,620; 6,523,964; 6,611,202;6,201,642; 6,690,268; 6,717,610; 6,757,109; 6,802,617; 6,806,452;6,822,563; 6,891,563; 6,946,978; 7,859,565; 5,550,677; 5,670,935;6,636,258; 7,145,519; 7,161,616; 7,230,640; 7,248,283; 7,295,229;7,301,466; 7,592,928; 7,881,496; 7,720,580; 7,038,577; 6,882,287;5,929,786 and/or 5,786,772, and/or U.S. Publication Nos.US-2014-0340510; US-2014-0313339; US-2014-0347486; US-2014-0320658;US-2014-0336876; US-2014-0307095; US-2014-0327774; US-2014-0327772;US-2014-0320636; US-2014-0293057; US-2014-0309884; US-2014-0226012;US-2014-0293042; US-2014-0218535; US-2014-0218535; US-2014-0247354;US-2014-0247355; US-2014-0247352; US-2014-0232869; US-2014-0211009;US-2014-0160276; US-2014-0168437; US-2014-0168415; US-2014-0160291;US-2014-0152825; US-2014-0139676; US-2014-0138140; US-2014-0104426;US-2014-0098229; US-2014-0085472; US-2014-0067206; US-2014-0049646;US-2014-0052340; US-2014-0025240; US-2014-0028852; US-2014-005907;US-2013-0314503; US-2013-0298866; US-2013-0222593; US-2013-0300869;US-2013-0278769; US-2013-0258077; US-2013-0258077; US-2013-0242099;US-2013-0215271; US-2013-0141578 and/or US-2013-0002873, which are allhereby incorporated herein by reference in their entireties.

Changes and modifications in the specifically described embodiments canbe carried out without departing from the principles of the invention,which is intended to be limited only by the scope of the appendedclaims, as interpreted according to the principles of patent lawincluding the doctrine of equivalents.

1. A display system for a vehicle, said display system comprising: aninterior camera disposed in an interior cabin of a vehicle and having afield of view interior of the vehicle that encompasses an area at ornear a user input for an accessory, and wherein the user input isaccessible by a driver of the vehicle when the driver is normallyoperating the vehicle; a display device viewable by the driver when thedriver is normally operating the vehicle, wherein said display device isoperable to display images derived from image data captured by saidinterior camera; and wherein said display device displays imagesrepresentative of the user input and of the driver's hand as thedriver's hand approaches the user input for the accessory for actuatingthe user input to adjust or control the accessory.
 2. The display systemof claim 1, wherein said display device displays images of the driver'shand as captured by said interior camera.
 3. The display system of claim2, wherein the displayed images of the driver's hand are partiallytransparent so the displayed images representative of the user input areviewable through the displayed images of the driver's hand.
 4. Thedisplay system of claim 1, wherein said display device displays imagesof an avatar of the driver's hand.
 5. The display system of claim 1,wherein said display device displays images representative of portionsof the driver's hand.
 6. The display system of claim 1, wherein saiddisplay device comprises a head up display that displays images forviewing by the driver when the driver is looking at and through thewindshield of vehicle.
 7. The display system of claim 1, comprising animage processor operable to process image data captured by said interiorcamera, and wherein said display device displays images responsive toprocessing by said image processor of image data captured by saidinterior camera.
 8. The display system of claim 1, wherein the displayedimages representative of the driver's hand are partially transparent sothe displayed images representative of the user input are viewablethrough the displayed images of the driver's hand.
 9. The display systemof claim 1, wherein the displayed images representative of the userinput are stored images of the user input that are merged with displayedimages representative of the driver's hand at the user input.
 10. Thedisplay system of claim 1, wherein the displayed images representativeof the user input are images of the user input that are captured by saidinterior camera.
 11. A display system for a vehicle, said display systemcomprising: an interior camera disposed in an interior cabin of avehicle and having a field of view interior of the vehicle thatencompasses an area at or near a user input for an accessory, andwherein the user input is accessible by a driver of the vehicle when thedriver is normally operating the vehicle; a display device viewable bythe driver when the driver is normally operating the vehicle, whereinsaid display device is operable to display images derived from imagedata captured by said interior camera; wherein said display devicecomprises a head up display that displays images for viewing by thedriver when the driver is looking at and through the windshield ofvehicle; wherein said display device displays images representative ofthe user input and of the driver's hand as the driver's hand approachesthe user input for the accessory for actuating the user input to adjustor control the accessory; and wherein the displayed images of the userinput are stored images of the user input that are merged with displayedimages representative of the driver's hand at the user input.
 12. Thedisplay system of claim 11, wherein said display device displays imagesof the driver's hand as captured by said interior camera, and whereinthe displayed images of the driver's hand are partially transparent sothe displayed images representative of the user input are viewablethrough the displayed images of the driver's hand.
 13. The displaysystem of claim 11, wherein said display device displays images of anavatar of the driver's hand.
 14. The display system of claim 11, whereinsaid display device displays iconistic representations of portions ofthe driver's hand.
 15. The display system of claim 11, wherein thedisplayed images representative of the driver's hand are partiallytransparent so the displayed images representative of the user input areviewable through the displayed images of the driver's hand.
 16. Adisplay system for a vehicle, said display system comprising: aninterior camera disposed in an interior cabin of a vehicle and having afield of view interior of the vehicle that encompasses an area at ornear a user input for an accessory, and wherein the user input isaccessible by a driver of the vehicle when the driver is normallyoperating the vehicle; a display device viewable by the driver when thedriver is normally operating the vehicle, wherein said display device isoperable to display images derived from image data captured by saidinterior camera; an image processor operable to process image datacaptured by said interior camera, and wherein said display devicedisplays images responsive to processing by said image processor ofimage data captured by said interior camera; wherein said display devicecomprises a head up display that displays images for viewing by thedriver when the driver is looking at and through the windshield ofvehicle; wherein said display device displays images representative ofthe user input and of the driver's hand as the driver's hand approachesthe user input for the accessory for actuating the user input to adjustor control the accessory; wherein the displayed images representative ofthe user input are stored images of the user input that are merged withdisplayed images representative of the driver's hand at the user input;and wherein the displayed images representative of the driver's hand areone of (i) partially transparent, (ii) an avatar of a person's hand, and(iii) iconistic representations of portions of the driver's hand. 17.The display system of claim 16, wherein the displayed imagesrepresentative of the driver's hand are partially transparent so thedisplayed images representative of the user input are viewable throughthe displayed images of the driver's hand.
 18. The display system ofclaim 17, wherein said display device displays images of the driver'shand as captured by said interior camera.
 19. The display system ofclaim 16, wherein said display device displays images of an avatar ofthe driver's hand.
 20. The display system of claim 16, wherein saiddisplay device displays iconistic representations of portions of thedriver's hand.